Determinacy and Decidability of Reachability 
Games with Partial Observation on Both Sides 

Nathalie Bertrand, Blaise Genest, Hugo Gimbert 
November 25, 2008 

Abstract 

We consider two-players stochastic reachability games with partial ob- 
servation on both sides and finitely many states, signals and actions. We 
prove that in such games, either player 1 has a strategy for winning with 
probability 1, or player 2 has such a strategy, or both players have strate- 
gies that guarantee winning with non-zero probability (positively winning 
strategies). We give a fix-point algorithm for deciding which of the three 
cases holds, which can be decided in doubly-exponential time. 

Introduction 

We prove two determinacy and decidability results about two-players stochastic 
reachability games with partial observation on both sides and finitely many 
states, signals and actions. Player 1 wants the play to reach the set of target 
states, while player 2 wants to keep away the play from target states. Players 
take their decisions based upon signals that they receive all along the play, but 
they cannot observe the actual state of the game, nor the actions played by 
their opponent, nor the signals received by their opponent. Each player only 
observes the signals he receives and the actions he plays. Players have common 
knowledge of the initial state of the game. 

Our determinacy result is of a special kind, it concerns two notions of so- 
lutions for stochastic games. The first one is the well known notion of almost- 
surely winning strategy, which guarantees winning with probability 1 against 
any strategy of the opponent. The second one is the notion of positively winning 
strategy: a strategy is positively winning if it guarantees a non-zero winning 
probability against any strategy of the opponent. This notion is less known, to 
our knowledge it appeared recently in [ Hor08| . The notion of positively winning 
strategy is different from the notion of positive value, because the non-zero win- 
ning probability can be made arbitrarily small by the opponent, hence existence 
of a positively winning strategy does not give any clue for deciding whether the 
value is zero or not. Existence of a positively winning strategy guarantees that 
the opponent does not have an almost-surely winning strategy, however there is 
no straightforward reason that one of these cases should always holds. Actually, 
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if we consider more complex classes of games than reachability games, there are 
various examples where neither player 1 has a positively winning strategy nor 
player 2 has an almost-surely winning strategy. 

Our first result (Theorem [J) states that, in reachability games with partial 
observation on both sides, either player 1 has a positively winning strategy or 
player 2 has an almost-surely winning strategy. Moreover which case holds is 
decidable in exponential time. Notice that an almost-surely winning strategy 
for player 2 in a reachability game is surely winning as well. 

Our second result (Theorem [3]) states that either player 1 has an almost- 
surely winning strategy or player 2 has a positively winning strategy, and this 
is decidable in doubly- exponential time. 

Both these results strengthen and generalize in several ways results given 
in [CDHROT] . Actually, in this paper is addressed only the particular case where 
player 2 has perfect information and target states are observable by player 1. 
Moreover in [CDHROT] no determinacy result is established, the paper "only" 
describes an algorithm for deciding whether player 1 has an almost-sure winning 
strategy. 

1 Reachability games with partial observation 
on both sides 

We consider zero-sum stochastic games with partial observation on both sides, 
where the goal of Player 1 is to reach a certain set of target states. Players only 
partially observe the state of the game, via signals. Signals and state transitions 
are governed by probability transitions: when the state is k and two actions i 
and j are chosen, player 1 and 2 receive respectively signals c and d and the 
new state is I with probability p(c, d,l \ k, i, j). 



1.1 Notations 

We use the following standard notations [RenOOj . 

The game is played in steps. At each step the game is in some state k € K . 
The goal of player 1 is to reach target states T C K. Before the game starts, 
the initial state is chosen according to the initial distribution 5 G ^{K), which 
is common knowledge of both players. Players 1 and 2 choose actions i G I 
and j € J, then player 1 receives a signal c G C, player 2 receives a signal 
d E D, and the game moves to a new state I. This happens with probability 
p{c,d,l I k,i,j) given by fixed transition probabilities p : K x I x J ^ V{C x 
D X K), known by both players. We denote p{l \ k,i,j) = J2c dPi''^'^'^ 
k,i,j). Players observe and remember their own actions and the signals they 
receive, it is convenient to suppose that in the signal they receive is encoded 
the action they just played, formally their exists act : C U D — > / U J such that 
p{c,d,k' I k,i,j) > {i — act(c) and j — act{d)). We denote p{c,d,l 

k) = p{c,d,l I fc, act(i), act(j)). This way, plays can be described by sequences 
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of states and signals for both players, without mentioning which actions were 
played. A sequence p = (fco, ci, di, . . . , c„, c?„, kn) G {KCD)*K is a finite play if 
for every < m < n, p{cm+i,dm+i,km+i \ act(cm+i), act(d„+i)) > 0. An 
infinite play is a sequence p G (KCD)'^ whose prefixes are finite plays. 

A strategy of player 1 is a mapping a : T>{K) x C* ^ a-nd a strategy 
of player 2 is r : V{K) x D* ^ V[J). 

In the usual way, an initial distribution 5 and two strategies a and r define 
a probability measure P^''' (•) on the set of infinite plays, equipped with the 
cr-algebra generated by cylinders. 

We use random variables Kn, In, Jn,Cn, Dn for designing respectively the 
n-th state, action of player 1, action of player 2, signal of player I, signal of 
player 2. The probability to reach a target state someday is: 

7i(5,a,r)=P^-"(3meN,i^„, gT) , 

and the probability to never reach the target is 72((5, cr, r) = I — 72((5, cr, r). 
Player 1 seeks maximizing 71 while player 2 seeks maximizing 72. 

1.2 Winning almost-surely or positively 

Definition 1 (Almost-surely and positively winning). A distribution S is almost- 
surely winning for player 1 if there exists a strategy a such that 

Vr,7i((5,a,T) = I . (1) 

A distribution 6 is positively winning for player 1 if there exists a strategy a 
such that 

Vr,7i(,5,a,r) >0 . (2) 

If the uniform distribution on a set of states L C_ K is almost-surely or positively 
winning then L itself is said to be almost-surely or positively winning. If there 
exists a such that ([T]) holds for every almost-surely winning distribution then a 
is said to be almost-surely winning . 

Positively winning strategies for player 1 and almost-sure winning and pos- 
itively winning strategies for player 2 are defined similarly. 

2 Winning almost-surely and positively with fi- 
nite memory 

Of special algorithmic interest are strategies with finite memory. 

Definition 2 (Strategies with finite memory). A strategy a with finite memory 
is described by a finite set M called the memory, a strategic function um '■ M ^ 
T>{I), an update function updatCjy^ : M x C M , an initialization function 
initA/ : V{K) M. 

For playing with a , player I proceeds as follows. Let 6 be the initial distribu- 
tion with support L, then initially player 1 puts the memory in state initM(-^). 



3 



When the memory is in state m, player 1 chooses his action according to the 
distribution ctm('t^)- When player 1 receives a signal c and its memory state is 
m, he changes the memory state to update^,j(TO, c). 

A crucial tool for establishing decidability and determinacy result is the class 
of finite memory strategy whose finite memory if based on the notions of beliefs 
or pessimistic beliefs. 

2.1 Beliefs and pessimistic beliefs 

The belief of a player at some moment of the play is the set of states he thinks 
the game could possibly be, according to the signals he received up to now. The 
pessimistic belief is similar, except the player assumes that no final state has 
been reached yet. One of the motivations for introducing beliefs and pessimistic 
beliefs is Proposition [1] 

Beliefs of player 1 are defined by mean of the operator Bi that associates 
with L C K and c G C, 

Bi{L, c) ^ {k e K \ 3l e L,3de D,p{k, c, d I > 0} . (3) 

We defined inductively the belief after signals Ci, . . . , c„ by Ci, . . . , c„, c) = 

Bi{Bi{L,ci, . . . ,c„),c). 

Pessimistic beliefs of player 1 are defined by 

B{{L,c)=Bi{L\T,c) . 

Beliefs B2 and pessimistic beliefs B2 for player 2 are defined similarly. We 
will use the following properties of beliefs and pessimistic beliefs. 

Proposition 1. Let a, r be strategies for player 1 and 2 and 6 an initial distri- 
bution with support L. Then for every n G N, 

P^g-^ {Kn+i e Bi{L,Ci, . . . ,Cn)) = 1 , 
F^s'" {Kn+ieB2{L,Di,...,D,,)) = l , 

P^'^ {Kn+i G Bl{L, Ci, . . . , C„) or Kra^T for some 1 < m < n) = 1 , 
P^'^ {K,,+i G Bl{L, £>!,..., AO or G T for some 1 < to < n) = 1 . 

Suppose T and S almost-surely winning for player 2, then for every n G N, 

P^'^ (■^2(^7 Di, . . . , Dn) is a.s.w. for player 2) = 1 . 

Suppose a and 6 almost surely winning for player 1, then for every n G N, 

P^'^ {B\{L, Ci, . . . , Cn) is a.s.w. for player 1 or 31 < to < n. Km G T) = 1 . 

Proof. Almost straightforward from the definitions. □ 
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2.2 Belief and pessimistic belief strategies 

A strategy a is said to be a belief strategy for player 1 if it has finite memory 
M = V{K) and 

1. the initial state of the memory is the support of the initial distribution, 

2. the update function is {L, c) t3i{L, c), 

3. the strategic function V{K) T^il) associates with each memory state 
L <Z K the uniform distribution on a non-empty set of actions II C /. 

The definition of a pessimistic belief strategy for player 1 is the same, except 
the update function is B^. 

3 Determinacy and decidability results 

In this section, we establish our main result, a determinacy result of a new kind. 
Usual determinacy results in game theory concern the existence of a value. Here 
the determinacy refers to positive and almost-sure winning: 

Theorem 1 (Determinacy). Every initial distribution is either almost- surely 
winning for player 1, surely winning for player 2 or positively winning for both 
players. 

Theorem [T] is a corollary of Theorems [5] and [31 in which details are given 
about the complexity of deciding whether an initial distribution is positively 
winning for player 1 and whether it is positively winning for player 1. 

Deciding whether a distribution is positively winning for player 1 is quite 
easy, because player 1 has a very simple strategy for winning positively: playing 
randomly any action. 

Theorem 2 (Deciding positive winning for player 1). Every initial distribution 
is either positively winning for player 1 or surely winning for player 2. 

The strategy for player 1 which plays randomly any action is positively win- 
ning. Player 2 has a belief strategy which is surely winning. 

The partition of supports between those positively winning for player 1 and 
those surely winning for player 2 is computable in time exponential in \K\, 
together with an almost- surely winning belief strategy for player 2. 

Proof of Theorem\^ Let £oo C 'P{K\T) be the greatest fix-point of the mono- 
tonic operator $ : r{V{K\T)) ViPiKXT)) defined by: 

$(£) = {L e C\3j e J,yde D, \i j = act(d) then B2{L,d) e £}, 

and let an be the strategy for player 1 that plays randomly any action. To 
establish Theorem [2] we are going to prove that: 

(A) every support in Coo is surely winning for player 2, and 
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(B) an is positively winning from any support L C K which is not in £00. 

We start with proving (A). For winning surely from any support L G £00, 
player 2 uses the following belief strategy: if the current belief of player 2 is 
L £ £00 then player 2 chooses an action Jl such that whatever signal d player 
2 receives (with act{d) — jl), his next belief B2[L, d) will be in C^o as well. By 
definition of $ there always exists such an action j, and this defines a belief- 
strategy a : L ^ Jl for player 2. When playing with this strategy, beliefs of 
player 2 never intersect T hence according to Proposition[Tl against any strategy 
a of player 1, the play stays almost-surely in K\T, hence it stays surely in K\T. 

Conversely, we prove (B). We fix the strategy for player 1 which consists 
in playing randomly any action with equal probability, and the game is a one- 
player game where only player 2 has choices to make: it is enough to prove (B) 
in the special case where the set of actions of player 1 is a singleton / = {i}. 
Let Cq = V{K\T) D Ci = $(£0) ^ £2 = ^(Ci) ... and Coc be the limit of this 
sequence, the greatest fixpoint of <f>. We prove that for any support L E V{K), 
\i L ^ Coo then: 

L is positively winning for player 1 . (4) 

If L n T ^ 0, (g]) is obvious. For delaing with the case where L e V{K\T), we 
define for every n e N, /C^ = 'P{K\T)\Cn, and we prove by induction on n G N 
that for every L G /C^, then for every initial distribution 5^ with support L, for 
every strategy r, 

PJ^ (3m G N, X„ G T, 2 < TO < n + 1) > . (5) 

For n = 0, (O is obvious because ICq — 0. Suppose that for some n G N, 
(O holds for every L G /C„, and let L G /Cn+i. If L G /C„ then by induc- 
tive hypothesis, ([5]) holds. Otherwise by definition of /C„+i, L G £„\<I'(£„) 
hence by definition of $, whatever action j is played by player 2 at the first 
round, there exists a signal dj such that a,ct{dj) ~ j and B2{L,dj) ^ £„. 
Let T be a strategy for player 2 and j an action such that r((5i)(j) > 0. If 
B2{L,dj) n T ^ then according to Proposition [J PJ^ {K2 G T) > 0. Other- 
wise B2{L, dj) G V{K\T)\Cn = ICn hence according to the inductive hypothesis 
^'B2{Ld) eN,2 <m <n + l,Km eT) > 0. Since player 1 has only one 
action, by definition of beliefs, for every state I G B2{Ld,j), PJ^ {K2 ~ I) > 0. 
Together with the previous equation, we obtain 

PJ^ (3to G N, 3 < m < n + 2, Km G T) > 0. This achieves the inductive step. 

The computation of the partition of supports between those positively win- 
ning for player 1, and those surely winning for player 2 and a surely winning 
strategy for player 2 amounts to the computation of the largest fixpoint of 
since $ is monotonic, and each application of the operator can be computed in 
exponential time, the overall computation can be achieved in exponential time 
and space. □ 

Deciding whether an initial distribution is positively winning for player 1 is 
easy because player 1 has a very simple strategy for that: playing randomly. 
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Figure 1: A game where player 2 needs a lot of memory. 



Player 2 does not have such a simple strategy for winning positively: he has 
to make hypotheses about the beliefs of player 1, as is shown in the example 
depicted by fig. [TJ 

Theorem 3 (Deciding positive winning for player 2). Every initial distribution 
is either almost-surely winning for player 1 or positively winning for player 2. 

Player 1 has an almost-surely winning strategy which is pessimistic belief. 
Player 2 has a finite memory strategy such that each memory state is a pair of 
a state and a pessimistic belief of player 1. 

The partition of supports between those almost-surely winning for player 1 
and those positively winning for player 2 is computable in time doubly- exponential 
in \K\, together with the winning strategies for both players. 

The proof of Theorem [3] is based on the following intuition. The easiest way 
of winning for player 2 is to reach with positive probability a state from where 
he wins surely. Hence player 1 will try to prevent the play from reaching such 
surely winning states, in other words player 1 should prevent his pessimistic 
belief to contain such surely winning states. However, doing so, player 1 may 
prevent the play to reach a target state: it may hold that player 2 has a strategy 
for winning positively under the hypothesis that pessimistic beliefs of player 1 
never contains surely winning states. This adds new beliefs of player 1 to the 
collection of pessimistic beliefs that player 1 should avoid. And so on... 

For formalizing these intuitions, we make use of C-games. 

Definition 3 {C-games) . Let C ^ K be a collection of supports. The C-game 
associated with C is the game with same actions, transitions and signals than the 
original partial observation game, only the winning condition changes: player 
1 loses if either the play never reaches a target state or if at some moment 
the pessimistic belief of player 1 is in C and the play has never visited a target 
state previously. Formally given an initial distribution 5 with support L and two 
strategies a and r the winning probability of player 1 is: 

P^'^(3ri >l,KneT and\fm < n,Bi{L,Ci, . . . ,C,n) ^ C) . 

Actually £-games are special cases of reachability games, as shown in the 
next proposition and its proof. 

Proposition 2. In a C-game, every support is either positively winning for 
player 2 or almost-surely winning for player 1. This partition can be computed 
in time doubly- exponential in \K\. Player 2 has a positively winning strategy 
whose states are pairs of states and pessimistic beliefs of player 1 . Player 1 has 
an almost-surely winning pessimistic-belief strategy. 

Proof. We define a reachability game Gc associated with C in the following 
way. Make the synchronized product of the original game with pessimistic 
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beliefs of player 1: each state is a pair {k,L) with k G K and L C K\T. 
Transitions are inherited from the original game except that every state whose 
second component is in C is absorbing. The set of target states is the set of 
pairs whose first component is in T. According to Theorem [5] in the reachability 
game Gc every state is either positively winning for player 2 or almost-surely 
winning for player 1. Moreover according to Theorem[21 player 2 has a positively 
winning belief strategy tc in Gc from which it is easy to construct a positively 
winning strategy in the C-game, with finite memory, whose memory states are 
sets of states of Gc- Also according to Theorem^ player 1 has an almost-surely 
pessimistic belief strategy ac in Gc- Notice that pessimistic beliefs of player 

1 in Gc cannot take all the possible values in 'P{{K\T) x V{K\T)) because 
intuitively player 1 has perfect knowledge about his own pessimistic beliefs and 
formally such a pessimistic belief is always of the type Di^lHI, L)} for some 
L C K\T. As a consequence, it is easy to extract from ac a pessimistic belief 
almost-surely winning strategy in the £-game. □ 

The heart of the proof of Theorem [3] is based on the two next propositions. 

Proposition 3. Let C be an upward- closed collection of supports. Suppose that 
every support in C is positively winning for player 2 in the original game. 

Then any support positively winning for player 2 in the C-game is positively 
winning in the original game as well. 

If apart from supports in C, there are no supports positively winning for 
player 2 in the C-game, then every support L ^ C is almost-surely winning for 
player 1 in the original game. 

Proof. Let Cp be the set of supports positively winning in the £-game, that are 
not in C 

We start with the case where Cp is not empty. Let r be a strategy for player 

2 positively winning in the original game. Let r' be a strategy for player 2 
positively winning in the £-game. Let r" be the following strategy for player 2. 
Player 2 starts playing totally randomly any action with equal probability. At 
each step of the play, player 2 throws a dice with three sides to decide whether 
he should: 

• keep playing randomly, 

• pick randomly a support L E C, forget the past observations and switch 
definitively to strategy r with initial support L, 

• pick randomly a support L G Cp, forget the past observations and switch 
definitively to strategy r' with initial support L. 

Let us prove that r" is positively winning in the original game, i.e. for every 
strategy cr and initial distribution S with support L G Cp, 

P^'""(3n> 1,X„ eT) < 1 . (6) 
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By definition of r", there is non-zero probability that the play is consistent with 
r' i.e. 

^T" (V" > 1' Jn - r'{L, Di,..., Dn-i)) > . (7) 
Since r' is positively winning in the £-game, 

^T' (3*^ > 1, e r and Vm < n, Bi{L, d, . . . , Cm) ^ £) < 1 . (8) 

If P^'^' {3n > 1, Kn e T) < 1 then together with Q this gives dH). 

If P^'^' (3n > 1, Kn eT) = 1 then according to ©, there exists TV > 1 and a 

pessimistic belief B E C such that: 

P^'"'' (Si(L, Ci, . . . , Cat) = S and VI < m < iV, is:,„ T) > . 

Since every sequence of actions is played with positive probability by t" , then: 

e B; P^'^" [Kn = ; and VI < m < TV, ^ T) > . (9) 

By definition of r", there is positive probability that r" picks randomly the 
support B € C and switches to r with initial support B. By definition, r is 
positively winning from B hence there exists I G B such that: 

Va',Pp^(Vn > l,if„ ^T) > , 

together with ^ it proves V^'^" {Kn = / and Vm > 1, K,n ^ T) > hence ©. 

Now we consider the case where Cp is empty. According to Proposition [2l 
player 1 has a pessimistic belief strategy a which is almost-surely winning in 
the £-game from every support L ^ C. This ensures, for every S whose support 
is L € £, for every strategy r, 

P^'" (V71 > 1, BP{L, Ci, ... ,Cn) ^ C or 3m < n, e T) = 1 . (10) 

We start with proving for each L ^ C there exists € N such that for 
every strategy r, for every distribution 6 with support L, 

Fg'^ i3n<NL,Kn&T)>^ . (11) 

We suppose such an Nl does not exist and seek for a contradiction. Suppose for 
every N there exists r^r and Sn such that pT|) does not hold. We can suppose 
Tat is deterministic i.e. r^r ; D* — > J, and Sn converges to some distribution 
6, whose support is included in L. Using Koenig's lemma, it is easy to build a 
strategy r : D* — *■ J such that for infinitely many N, 

Taking the limit when N oo, we get: 

P^^"(3n> l,A-„ eT) < 1 , 
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which contradicts the fact that a is almost-surely winning from L, since the 
support of 5 is included in L. This proves the existence of Nl such that 
holds. 

Now let TV = maxjiVi \ L ^ C] and let a' be the pessimistic belief strategy 
for player 1 similar to cr, except every N steps the memory is reset, formally: 
cr'(I/)(ci, . . . ,c„) a{B{{L,ci, . . . ,C(^n/N)*N))ic{n/N)*N ' ■•£„). Then whatever 
be the strategy played by player 2, according to (fTO| as long as a target state 
is not reached, the memory of cr' will stay outside C. Then according to (HJ), 
when playing a', every N steps there is probability at least ^ to reach a target 
state, knowing that it was not reached before, hence there is probability of 
never reaching a target state. Consequently, a is almost-surely winning from 
any support L ^ C. □ 

Now we can prove Theorem |3l 

Proof of Theorem\^ Let Cq,Ci, . . . be the sequence defined by £o = and for 
every n e N, Cn+i ^ 'PiK) is the set of supports positively winning for player 
2 in the £„-game. Then Cq C £i C . . . and V{V{Ky) is finite hence there is a 
limit Coo to this sequence. 

Every Cn is upward-closed hence according to Proposition [3l every support 
in Coo is positively winning for player 2. Moreover, according to Proposition [51 
player 2 has a positively winning strategy with finite memory whose memory 
states are sets of pairs a state and a pessimistic belief of player 1. 

By definition of Coo, the only support positively winning in the £oo-game 
are in Coo- Hence according to Proposition [3] again, every support not in Coo 
is almost-surely winning for player 1. Moreover, according to Proposition [21 
player 1 has a pessimistic belief almost-surely winning strategy. 

The computation of Coo can be achieved in doubly-exponential time, be- 
cause according to Proposition [2] each step can be carried on in time doubly 
exponential in K and since the sequence {Cn)nen is monotonic its length is at 
most exponential in |iir|. □ 



Conclusion 

We considered stochastic reachability games with partial observation on both 
sides. We established a determinacy result: such a game is either almost-surely 
winning for player 1, surely winning for player 2 or positively winning for both 
players. Despite its simplicity, this result is not so easy to prove. Also we gave 
algorithms for deciding in doubly-exponential time which of the three cases hold. 

A natural question is whether these results extend are true for Biichi games 
as well? The answer is " partially" . 

One one hand, it is possible to prove that a game is either almost-surely 
winning for player 1 or positively winning for player 2 and to decide in doubly- 
exponential time which of the two cases hold. This can be done by techniques 
almost identical to the ones in this paper. 
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On the other hand, it was shown recently that the question "does player 1 
has a deterministic strategy for winning positively a Biichi game?" is undecid- 
able |BBG08j , even when player 1 receives no signals and player 2 has only one 
action. It is quite easy to see that "deterministic" can be removed from this 
question, without changing its answer. Hence the only hope for solving positive 
winning for Biichi games is to consider subclasses of partial observation games 
where the undecidability result fails, an interesting question. 
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